Content-Based Filtering: My Approach¶

By Mohammad Sayem Chowdhury

Welcome to my notebook on content-based recommendation systems. Here, I demonstrate how to build a content-based recommender using Python and pandas, sharing my own workflow and insights. All code, analysis, and commentary are my own.

Author: Mohammad Sayem Chowdhury¶

Objectives¶

  • Build a content-based recommendation system
  • Explore the logic and implementation in my own style
  • Share personal insights and workflow

Recommendation systems are a collection of algorithms used to recommend items to users based on information taken from the user. These systems have become ubiquitous, and can be commonly seen in online stores, movies databases and job finders. In this notebook, we will explore Content-based recommendation systems and implement a simple version of one using Python and the Pandas library.

Table of contents¶

  1. Acquiring the Data
  2. Preprocessing
  3. Content-Based Filtering

Acquiring the Data¶

To acquire and extract the data, simply run the following Bash scripts:
Dataset acquired from GroupLens. Lets download the dataset. To download the data, we will use !wget to download it from IBM Object Storage.
Did you know? When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: Sign up now for free

In [1]:
# !wget -O moviedataset.zip https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%205/data/moviedataset.zip --no-check-certificate
# print('unziping ...')
# !unzip moviedataset.zip

Now you're ready to start working with the data!

Preprocessing¶

First, let's get all of the imports out of the way:

In [2]:
#Dataframe manipulation library
import pandas as pd
#Math functions, we'll only need the sqrt function so let's import only that
from math import sqrt
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Now let's read each file into their Dataframes:

In [3]:
#Storing the movie information into a pandas dataframe
movies_df = pd.read_csv('ml-latest/movies.csv')

#Head is a function that gets the first N rows of a dataframe. N's default is 5.
movies_df.head()
Out[3]:
movieId title genres
0 1 Toy Story (1995) Adventure|Animation|Children|Comedy|Fantasy
1 2 Jumanji (1995) Adventure|Children|Fantasy
2 3 Grumpier Old Men (1995) Comedy|Romance
3 4 Waiting to Exhale (1995) Comedy|Drama|Romance
4 5 Father of the Bride Part II (1995) Comedy
In [4]:
#Storing the user information into a pandas dataframe
ratings_df = pd.read_csv('ml-latest/ratings.csv')
#Head is a function that gets the first N rows of a dataframe. N's default is 5.
ratings_df.head()
Out[4]:
userId movieId rating timestamp
0 1 169 2.5 1204927694
1 1 2471 3.0 1204927438
2 1 48516 5.0 1204927435
3 2 2571 3.5 1436165433
4 2 109487 4.0 1436165496

Let's also remove the year from the title column by using pandas' replace function and store in a new year column.

In [5]:
#Using regular expressions to find a year stored between parentheses
#We specify the parantheses so we don't conflict with movies that have years in their titles
movies_df['year'] = movies_df.title.str.extract('(\(\d\d\d\d\))',expand=False)
#Removing the parentheses
movies_df['year'] = movies_df.year.str.extract('(\d\d\d\d)',expand=False)
#Removing the years from the 'title' column
movies_df['title'] = movies_df.title.str.replace('(\(\d\d\d\d\))', '', regex=True)
#Applying the strip function to get rid of any ending whitespace characters that may have appeared
movies_df['title'] = movies_df['title'].apply(lambda x: x.strip())
movies_df.head()
Out[5]:
movieId title genres year
0 1 Toy Story Adventure|Animation|Children|Comedy|Fantasy 1995
1 2 Jumanji Adventure|Children|Fantasy 1995
2 3 Grumpier Old Men Comedy|Romance 1995
3 4 Waiting to Exhale Comedy|Drama|Romance 1995
4 5 Father of the Bride Part II Comedy 1995

With that, let's also split the values in the Genres column into a list of Genres to simplify future use. This can be achieved by applying Python's split string function on the correct column.

In [6]:
#Every genre is separated by a | so we simply have to call the split function on |
movies_df['genres'] = movies_df.genres.str.split('|')
movies_df.head()
Out[6]:
movieId title genres year
0 1 Toy Story [Adventure, Animation, Children, Comedy, Fantasy] 1995
1 2 Jumanji [Adventure, Children, Fantasy] 1995
2 3 Grumpier Old Men [Comedy, Romance] 1995
3 4 Waiting to Exhale [Comedy, Drama, Romance] 1995
4 5 Father of the Bride Part II [Comedy] 1995

Since keeping genres in a list format isn't optimal for the content-based recommendation system technique, we will use the One Hot Encoding technique to convert the list of genres to a vector where each column corresponds to one possible value of the feature. This encoding is needed for feeding categorical data. In this case, we store every different genre in columns that contain either 1 or 0. 1 shows that a movie has that genre and 0 shows that it doesn't. Let's also store this dataframe in another variable since genres won't be important for our first recommendation system.

In [7]:
#Copying the movie dataframe into a new one since we won't need to use the genre information in our first case.
moviesWithGenres_df = movies_df.copy()

#For every row in the dataframe, iterate through the list of genres and place a 1 into the corresponding column
for index, row in movies_df.iterrows():
    for genre in row['genres']:
        moviesWithGenres_df.at[index, genre] = 1
#Filling in the NaN values with 0 to show that a movie doesn't have that column's genre
moviesWithGenres_df = moviesWithGenres_df.fillna(0)
moviesWithGenres_df.head()
Out[7]:
movieId title genres year Adventure Animation Children Comedy Fantasy Romance ... Horror Mystery Sci-Fi IMAX Documentary War Musical Western Film-Noir (no genres listed)
0 1 Toy Story [Adventure, Animation, Children, Comedy, Fantasy] 1995 1.0 1.0 1.0 1.0 1.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 2 Jumanji [Adventure, Children, Fantasy] 1995 1.0 0.0 1.0 0.0 1.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 3 Grumpier Old Men [Comedy, Romance] 1995 0.0 0.0 0.0 1.0 0.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 4 Waiting to Exhale [Comedy, Drama, Romance] 1995 0.0 0.0 0.0 1.0 0.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 5 Father of the Bride Part II [Comedy] 1995 0.0 0.0 0.0 1.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

5 rows × 24 columns

Next, let's look at the ratings dataframe.

In [8]:
ratings_df.head()
Out[8]:
userId movieId rating timestamp
0 1 169 2.5 1204927694
1 1 2471 3.0 1204927438
2 1 48516 5.0 1204927435
3 2 2571 3.5 1436165433
4 2 109487 4.0 1436165496

Every row in the ratings dataframe has a user id associated with at least one movie, a rating and a timestamp showing when they reviewed it. We won't be needing the timestamp column, so let's drop it to save on memory.

In [9]:
#Drop removes a specified row or column from a dataframe
ratings_df = ratings_df.drop(labels = 'timestamp', axis = 1)
ratings_df.head()
Out[9]:
userId movieId rating
0 1 169 2.5
1 1 2471 3.0
2 1 48516 5.0
3 2 2571 3.5
4 2 109487 4.0

Content-Based recommendation system¶

Now, let's take a look at how to implement Content-Based or Item-Item recommendation systems. This technique attempts to figure out what a user's favourite aspects of an item is, and then recommends items that present those aspects. In our case, we're going to try to figure out the input's favorite genres from the movies and ratings given.

Let's begin by creating an input user to recommend movies to:

Notice: To add more movies, simply increase the amount of elements in the userInput. Feel free to add more in! Just be sure to write it in with capital letters and if a movie starts with a "The", like "The Matrix" then write it in like this: 'Matrix, The' .

In [10]:
userInput = [
            {'title':'Toy Story', 'rating':3.5},
            {'title':'Jumanji', 'rating':2},
            {'title':"Pulp Fiction", 'rating':5},
            {'title':'Breakfast Club, The', 'rating':2},
            {'title':'Prestige, The', 'rating':4.5},
            {'title':'Interstellar', 'rating':5},
            {'title':"Dark Knight, The", 'rating':5},
            {'title':'Akira', 'rating':2.5},
            {'title':'Batman Begins', 'rating':4.5},
            {'title':'Iron Man', 'rating':3.5},
            {'title':'Inception', 'rating':4.5}         ] 
inputMovies = pd.DataFrame(userInput)
inputMovies
Out[10]:
title rating
0 Toy Story 3.5
1 Jumanji 2.0
2 Pulp Fiction 5.0
3 Breakfast Club, The 2.0
4 Prestige, The 4.5
5 Interstellar 5.0
6 Dark Knight, The 5.0
7 Akira 2.5
8 Batman Begins 4.5
9 Iron Man 3.5
10 Inception 4.5

Add movieId to input user¶

With the input complete, let's extract the input movie's ID's from the movies dataframe and add them into it.

We can achieve this by first filtering out the rows that contain the input movie's title and then merging this subset with the input dataframe. We also drop unnecessary columns for the input to save memory space.

In [11]:
#Filtering out the movies by title
inputId = movies_df[movies_df['title'].isin(inputMovies['title'].tolist())]
#Then merging it so we can get the movieId. It's implicitly merging it by title.
inputMovies = pd.merge(inputId, inputMovies)
#Dropping information we won't use from the input dataframe
inputMovies = inputMovies.drop(labels = 'genres', axis = 1).drop(labels = 'year', axis =1)
#Final input dataframe
#If a movie you added in above isn't here, then it might not be in the original 
#dataframe or it might spelled differently, please check capitalisation.
inputMovies
Out[11]:
movieId title rating
0 1 Toy Story 3.5
1 2 Jumanji 2.0
2 296 Pulp Fiction 5.0
3 1274 Akira 2.5
4 1968 Breakfast Club, The 2.0
5 33794 Batman Begins 4.5
6 48780 Prestige, The 4.5
7 58559 Dark Knight, The 5.0
8 59315 Iron Man 3.5
9 79132 Inception 4.5
10 109487 Interstellar 5.0

We're going to start by learning the input's preferences, so let's get the subset of movies that the input has watched from the Dataframe containing genres defined with binary values.

In [12]:
#Filtering out the movies from the input
userMovies = moviesWithGenres_df[moviesWithGenres_df['movieId'].isin(inputMovies['movieId'].tolist())]
userMovies
Out[12]:
movieId title genres year Adventure Animation Children Comedy Fantasy Romance ... Horror Mystery Sci-Fi IMAX Documentary War Musical Western Film-Noir (no genres listed)
0 1 Toy Story [Adventure, Animation, Children, Comedy, Fantasy] 1995 1.0 1.0 1.0 1.0 1.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 2 Jumanji [Adventure, Children, Fantasy] 1995 1.0 0.0 1.0 0.0 1.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
293 296 Pulp Fiction [Comedy, Crime, Drama, Thriller] 1994 0.0 0.0 0.0 1.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1246 1274 Akira [Action, Adventure, Animation, Sci-Fi] 1988 1.0 1.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1885 1968 Breakfast Club, The [Comedy, Drama] 1985 0.0 0.0 0.0 1.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10176 33794 Batman Begins [Action, Crime, IMAX] 2005 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
11411 48780 Prestige, The [Drama, Mystery, Sci-Fi, Thriller] 2006 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
12540 58559 Dark Knight, The [Action, Crime, Drama, IMAX] 2008 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
12646 59315 Iron Man [Action, Adventure, Sci-Fi] 2008 1.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
15562 79132 Inception [Action, Crime, Drama, Mystery, Sci-Fi, Thrill... 2010 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
23044 109487 Interstellar [Sci-Fi, IMAX] 2014 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0

11 rows × 24 columns

We'll only need the actual genre table, so let's clean this up a bit by resetting the index and dropping the movieId, title, genres and year columns.

In [13]:
#Resetting the index to avoid future issues
userMovies = userMovies.reset_index(drop=True)
#Dropping unnecessary issues due to save memory and to avoid issues
userGenreTable = userMovies.drop(labels = 'movieId', axis = 1).drop(labels = 'title', axis = 1).drop(labels ='genres', axis = 1).drop(labels = 'year', axis = 1)
userGenreTable
Out[13]:
Adventure Animation Children Comedy Fantasy Romance Drama Action Crime Thriller Horror Mystery Sci-Fi IMAX Documentary War Musical Western Film-Noir (no genres listed)
0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 1.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 1.0 1.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
6 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
8 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 0.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0

Now we're ready to start learning the input's preferences!

To do this, we're going to turn each genre into weights. We can do this by using the input's reviews and multiplying them into the input's genre table and then summing up the resulting table by column. This operation is actually a dot product between a matrix and a vector, so we can simply accomplish by calling Pandas's "dot" function.

In [14]:
inputMovies['rating']
Out[14]:
0     3.5
1     2.0
2     5.0
3     2.5
4     2.0
5     4.5
6     4.5
7     5.0
8     3.5
9     4.5
10    5.0
Name: rating, dtype: float64
In [15]:
#Dot produt to get weights
userProfile = userGenreTable.transpose().dot(inputMovies['rating'])
#The user profile
userProfile
Out[15]:
Adventure             11.5
Animation              6.0
Children               5.5
Comedy                10.5
Fantasy                5.5
Romance                0.0
Drama                 21.0
Action                20.0
Crime                 19.0
Thriller              14.0
Horror                 0.0
Mystery                9.0
Sci-Fi                20.0
IMAX                  19.0
Documentary            0.0
War                    0.0
Musical                0.0
Western                0.0
Film-Noir              0.0
(no genres listed)     0.0
dtype: float64

Now, we have the weights for every of the user's preferences. This is known as the User Profile. Using this, we can recommend movies that satisfy the user's preferences.

Let's start by extracting the genre table from the original dataframe:

In [16]:
#Now let's get the genres of every movie in our original dataframe
genreTable = moviesWithGenres_df.set_index(moviesWithGenres_df['movieId'])
#And drop the unnecessary information
genreTable = genreTable.drop(labels = 'movieId', axis = 1).drop(labels = 'title', axis = 1).drop(labels = 'genres', axis = 1).drop(labels = 'year', axis = 1)
genreTable.head()
Out[16]:
Adventure Animation Children Comedy Fantasy Romance Drama Action Crime Thriller Horror Mystery Sci-Fi IMAX Documentary War Musical Western Film-Noir (no genres listed)
movieId
1 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 1.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 1.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
In [17]:
genreTable.shape
Out[17]:
(34208, 20)

With the input's profile and the complete list of movies and their genres in hand, we're going to take the weighted average of every movie based on the input profile and recommend the top twenty movies that most satisfy it.

In [18]:
#Multiply the genres by the weights and then take the weighted average
recommendationTable_df = ((genreTable*userProfile).sum(axis=1))/(userProfile.sum())
recommendationTable_df.head()
Out[18]:
movieId
1    0.242236
2    0.139752
3    0.065217
4    0.195652
5    0.065217
dtype: float64
In [19]:
#Sort our recommendations in descending order
recommendationTable_df = recommendationTable_df.sort_values(ascending=False)
#Just a peek at the values
recommendationTable_df.head()
Out[19]:
movieId
79132     0.757764
5018      0.686335
26701     0.677019
115479    0.655280
81132     0.652174
dtype: float64

Now here's the recommendation table!

In [20]:
#The final recommendation table
movies_df.loc[movies_df['movieId'].isin(recommendationTable_df.head(20).keys())]
Out[20]:
movieId title genres year
196 198 Strange Days [Action, Crime, Drama, Mystery, Sci-Fi, Thriller] 1995
515 519 RoboCop 3 [Action, Crime, Drama, Sci-Fi, Thriller] 1993
2900 2985 RoboCop [Action, Crime, Drama, Sci-Fi, Thriller] 1987
4923 5018 Motorama [Adventure, Comedy, Crime, Drama, Fantasy, Mys... 1991
9000 26701 Patlabor: The Movie (Kidô keisatsu patorebâ: T... [Action, Animation, Crime, Drama, Film-Noir, M... 1989
10427 37462 Wedlock [Action, Crime, Drama, Sci-Fi, Thriller] 1991
12720 59844 Honor Among Thieves (Adieu l'ami) (Farewell, F... [Action, Adventure, Crime, Drama, Mystery, Thr... 1968
12873 60684 Watchmen [Action, Drama, Mystery, Sci-Fi, Thriller, IMAX] 2009
13250 64645 The Wrecking Crew [Action, Adventure, Comedy, Crime, Drama, Thri... 1968
13552 67070 Army of One (Joshua Tree) [Action, Adventure, Crime, Drama, Mystery, Thr... 1993
14397 71999 Aelita: The Queen of Mars (Aelita) [Action, Adventure, Drama, Fantasy, Romance, S... 1924
15001 75408 Lupin III: Sweet Lost Night (Rupan Sansei: Swe... [Action, Animation, Comedy, Crime, Drama, Myst... 2008
15073 76153 Lupin III: First Contact (Rupan Sansei: Faasut... [Action, Animation, Comedy, Crime, Drama, Myst... 2002
15562 79132 Inception [Action, Crime, Drama, Mystery, Sci-Fi, Thrill... 2010
16055 81132 Rubber [Action, Adventure, Comedy, Crime, Drama, Film... 2010
16884 85261 Mars Needs Moms [Action, Adventure, Animation, Children, Comed... 2011
17152 86644 Fast Five (Fast and the Furious 5, The) [Action, Crime, Drama, Thriller, IMAX] 2011
21403 103651 Tai Chi Hero [Action, Comedy, Drama, Fantasy, Sci-Fi, IMAX] 2012
24565 115479 Whip Hand, The [Action, Adventure, Crime, Drama, Sci-Fi, Thri... 1951
26442 122787 The 39 Steps [Action, Adventure, Comedy, Crime, Drama, Thri... 1959

Advantages and Disadvantages of Content-Based Filtering¶

Advantages¶
  • Learns user's preferences
  • Highly personalized for the user
Disadvantages¶
  • Doesn't take into account what others think of the item, so low quality item recommendations might happen
  • Extracting data is not always intuitive
  • Determining what characteristics of the item the user dislikes or likes is not always obvious

This notebook and all analysis were created by Mohammad Sayem Chowdhury as a personal data science showcase.

Thank you for exploring my approach to content-based recommendation systems! If you have any feedback or suggestions, feel free to reach out.